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Abstract 


The  critical  challenge  addressed  by  this  research  task  is  the  development  of  systems  engineering  (SE)  talent  for  future  DoD  and  related 
industry  workforce  needs.  A  45%  growth  is  expected  in  SE  jobs  in  the  next  decade  ^  numerous  studies  have  highlighted  the  shortfalls 
in  both  the  number  and  capability  of  the  SE  workforce.  New  academic  and  career  pathways  are  urgently  needed  to  build  the  talent 
base  required  by  DoD  and  associated  industries.  Operating  through  the  Department  of  Defense  Research  and  Engineering  (DDR&E) 
STEM  Development  Office  and  the  Systems  Engineering  Research  Center  (SERC),  DoD  is  sponsoring  a  research  effort  aimed  at 
building  education  and  workforce  capacity  in  systems  engineering.  This  research  will  analyze  the  outcomes  on  student  learning  of  and 
career  interest  in  SE  and  DoD  careers  through  analysis  of  a  series  of  systems  engineering  pilot  capstone  courses.  These  courses  are 
being  offered  in  a  variety  of  contexts  at  eight  universities  and  separately,  through  six  service  academies.  Products  of  this  research  will 
include:  SE  curriculum  and  assessments;  compilation  and  analysis  of  the  methodologies  and  approaches  to  SE  course  delivery;  and 
recommendations  of  effective  practices  for  scaling  up  the  program  to  larger  numbers  of  institutions  nationwide. 


^  Rosato,  D.,  Braverman,  B.,  &  Jeffries,  A.  (2009,  November).  The  50  best  jobs  in  America.  Money,  88-96. 
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1  Summary 


This  research  effort  was  catalyzed  by  the  Department  of  Defense’s  challenge  to  increase  workforce  capacity  and  career  interest  in 
systems  engineering  (SE),  with  particular  focus  on  increasing  awareness  and  interest  among  undergraduate  and  graduate  students  of 
relevant,  engaging,  DoD  problems  and  careers  pathways.  The  objective  of  the  research  is  to  explore,  analyze,  and  make 
recommendations  based  on  a  variety  of  approaches  to  increase  undergraduate  and  graduate  learning  and  career  interest  in  SE  and 
related  DoD  careers.  Eunding  for  this  project  is  provided  by  the  Department  of  Defense  Research  and  Engineering  (DDR&E)  STEM 
Development  Office. 

The  project  has  three  phases: 

Phase  1/Startup  (March  1,  2010-May  15,  2010)  provided  program  requirements  and  subcontracts  to  enable  partner  universities  to 
develop  materials  and  conduct  program  implementation  in  the  Eall  2010  academic  semester.  Pilot  programs  are  one  or  two  semesters 
long.  Phase  1  had  three  key  activities: 

1 .  Develop  Requirements  and  Specifications  for  Capstone  Team  Projects 

2.  Develop  Draft  Capstone  Team  Project  Specifications  Document 

3.  Select  up  to  10  Capstone  Teams,  eight  from  SERC  university  partners  and  two  team  members  (AEIT  and  NPS) 
implementing  Capstone  Projects  and  coordinating  activities  among  service  academies. 

Contract  Number:  H98230-08-D-01 71 ,  D02,  TT02,  RT#01 9 

Report  No.  SERC-TR-008 
August  2,  2010 

UNCLASSIFIED 

7 


UNCLASSIFIED 


Phase  2/Pilot  Implementation  (May  15,  2010-June  30,  2011):  Capstone  Team  Members  develop  course  materials  and  assessment 
instruments;  deliver  the  courses  and  administer  assessments;  and  submit  two  interim  reports  and  a  final  report  that  assesses  the  extent 
to  which  the  learning  goals  were  met.  Phase  2(a)  included  planning  and  course  development;  development  of  assessment  frameworks 
and  instruments;  and  coordination  with  DoD  on  the  assignment  of  advisors  to  Capstone  Project  teams;  Phase  2(b)  includes  the 
delivery  of  courses  and  assessments. 

Phase  3/Analysis,  Recommendations  &  Dissemination  (July  1,  2011  -  September  30,  2011): 

Phase  3  will  analyze  the  results  from  all  Capstone  Team  Members  and  integrate  them  into  a  single  set  of  observations  to  the  sponsor 
about  the  effectiveness  of  the  pilot  programs,  analysis  of  pre-/post  learning  of  SE  content,  skills,  and  career  interest,  and  the  degree  to 
which  learning  outcomes  were  achieved.  It  will  also  create  specific  recommendations  on  how  to  scale-up  the  pilot  program  to  be 
conducted  across  the  U.S 

Following  the  development  and  release  of  a  request  for  proposals  (REP),  developed  in  consultation  with  the  DDR&E  STEM 
Development  Office,  the  following  11  SERC  institutions  submitted  proposals  for  RT-19.  A  comprehensive  review  by  an  independent 
panel  according  to  a  rubric  (See  Appendix  A)  led  to  the  recommendation  for  funding  eight  of  the  institutions  listed  below,  listed  in 
bold: 

•  Auburn  University 

•  Missouri  University  of  Science  and  Technology 

•  Pennsylvania  State  University 

•  Purdue  University 

•  Southern  Methodist  University 
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•  Stevens  Institute  of  Technology 

•  University  of  Alabama 

•  University  of  Maryland 

•  University  of  Southern  California 

•  University  of  Virginia 

•  Wayne  State  University 

In  addition,  six  service  academies/institutions  received  separate  funding  to  conduct  SE  Capstone  Projects;  coordinating  institutions  are 
noted  with  an  asterisk: 

•  Air  Force  Institute  of  Technology* 

•  Air  Force  Academy 

•  Coast  Guard  Academy 

•  Naval  Post  Graduate  School* 

•  U.S.  Naval  Academy 

•  U.S.  Military  Academy 

This  interim  report  will  focus  on  work  completed  through  July  31,  2010  (end  of  Phase  2(a)). 
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2  Progress  to  Date 


2.1  Objective  of  RT-19:  Research  on  Building  Education  and  Workforce  Capacity  in  Systems  Engineering 

The  objective  of  the  research  is  to  explore,  analyze,  and  make  recommendations  for  scale-up  based  on  a  variety  of  approaches  to 
increase  undergraduate  and  graduate  learning  and  career  interest  in  SE  and  related  DoD  careers.  Detailed  monthly  reports  have 
delineated  progress  toward  key  benchmarks.  Major  milestones  are  listed  below: 

Phase  1/Startup  (March  1,  2010-May  15,  2010) 

a.  RT-19  Invitation  to  SERC  Partners  was  completed,  disseminated,  and  shared  with  SERC  institutions  via  a  webex  technical 
assistance  session.  Invitation  included  project  requirements  and  specifications,  including  the  selection  of  one  of  five  DoD 
problems,  the  participation  of  a  DoD  mentor/client,  and  the  development  by  student  teams  of  a  real  product  or  artifact.  This 
invitation  included  as  Appendix  B. 

b.  RT-19  proposal  scoring  rubric  finalized;  external  review  panel  recruited. 

c.  Technical  assistance  session  detailing  review  process,  deadlines,  and  scoring  rubric  held  for  independent  review  panel, 
recorded  and  posted  on  project  website  on  Sakai. 

d.  11  SERC  member  institutions  submitted  proposals. 

e.  A  password  access  Sakai  site  was  created  to  house  proposals,  review  rubrics,  recorded  technical  assistance  webex  and  other 
key  project  artifacts. 

f.  8  proposals  recommended  for  funding.  These  institutions  and  the  principal  investigators  appear  as  Table  1. 
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Table  i. 


No 

Partner  Institution 

Principal  Investigator 

1 

Auburn  University 

David  Umphress,  PhD 

2 

Missouri  University  of  Science  and  Technology 

Cihan  Dagli,  PhD 

3 

Pennsylvania  State  University 

James  Nemes,  PhD 

4 

Southern  Methodist  University 

Liguo  Huang,  PhD 

5 

Stevens  Institute  of  Technology 

Keith  Sheppard,  PhD 

6 

University  of  Maryland 

John  Baras,  PhD 

7 

University  of  Virginia 

William  Sherer,  PhD 

8 

Wayne  State  University 

Walter  Bryzik,  PhD 

Note:  Air  Force  Institute  of  Technology  (AFIT)  and  Naval  Postgraduate  School  (NPS)  will 
coordinate  the  work  of  the  service  academies  in  the  implementation  of  the  Capstone  Project. 


g.  Submitting  principal  investigators  were  notified  of  the  status  of  the  funding  recommendations  (recommended  for  funding  or 
not  recommended  for  funding).  Those  recommended  for  funding  received  reviewer  comments  to  strengthen  their  proposals, 
while  those  not  recommended  for  funding  received  a  summary  of  reviewer  comments  (see  Appendix  C). 

h.  Sakai  site  and  access  was  established  for  DOD/AFIT/NPS 
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i.  Coordination  was  begun  between  DOD  representative  (B.  McGahem)  and  Capstone  team  Pis  to  connect  DOD  clients  with 
Capstone  project  teams. 

j.  May  26  SERC  Monthly  Collaborator  WebEx  meeting  updated  progress  of  RT-19 

k.  Pis  of  proposals  recommended  for  funding  were  notified  that  an  interim  report  will  be  due  on  7/15/2010,  which  is  to  include  an 
updated  workplan  and  timeline  that  incorporates  recommendations  from  the  review  panel  and  an  update  on  development  and 
implementation  status  as  of  that  date. 

Phase  2  (a)  May  15  -  July  31,  2010: 

The  following  activities  have  been  completed  or  are  in  progress  in  Phase  2(a): 

a.  Each  Capstone  Team  Member  submitted  an  updated  work-plan  and  timeline  that  incorporates  recommendations  from  the 
review  panel  (completed).  A  summary  of  the  curriculum  development,  course  revisions,  and  course  implementation  timeline 
from  each  Capstone  partner  appear  as  Appendix  D. 

b.  Each  Capstone  Team  Member  will  submit  a  comprehensive  interim  report  of  implementation  status  (completed).  Interim 
reports  from  each  partner  institution  contain  detailed  information  of  project  revisions  and  progress.  These  reports  are  available 
for  review  in  Sakai  at  the  following  link:  https://gatewav.stevens.edu/portal. 

c.  The  PEResearch  Team  will  compile  and  submit  a  project  report  comprised  of  updated  work-plan/timeline  and  interim  reports 
from  each  Capstone  Team  Member  (completed). 
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d.  Development  of  evaluation  framework  and  eommon  assessments  for  use  by  all  Capstone  Projects  (in  progress).  Section  3  details 

the  evaluation  and  assessment  work  in  progress,  and  Appendix  F  contains  the  evaluation  framework  and  draft  evaluation 
instruments. 

e.  Development  of  public  web  site  template  (in  progress).  Section  3.2  contains  a  draft  web  site  template,  subject  to  sponsor  input 

and  approval. 

f.  Coordination  with  AFIT  and  NPS  on  service  academies’  implementation  of  SE  Capstone  projects  (in  progress). 

g.  Review  of  Institutional  Review  Board  needs  and  status  for  all  Capstone  Team  partners  (in  progress). 

3  Evaluation  Framework  and  Assessments 


3.1  Evaluation  Framework 

The  evaluation  process  of  the  SE  Capstone  project  is  designed  to  understand  the  approaches,  relationships,  methodologies,  and 
curricular  materials  that  lead  to  increased  student  learning  of  and  career  interest  in  systems  engineering,  with  particular  emphasis  on 
DoD  problems  and  careers.  To  this  end,  data  will  be  collected  from  students,  from  faculty,  and  from  DoD  advisors  to  inform  the 
research  team.  Among  these  data  are: 

•  Reports  from  faculty  of  each  participating  institutions 

•  Surveys  of  participating  students,  faculty,  and  DoD  advisors 
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•  Ongoing  blogs  contributed  by  participating  students  during  the  conduct  of  the  courses 

•  Pre-  and  post-assessments  of  student  interest  in  SE,  understanding  of  SE,  and  ability  to  use  SE  concepts  and  approaches 

•  SE  case  study  analysis 

In  addition  to  these  project-wide  evaluation  instruments,  each  Capstone  Project  will  administer  and  report  on  outcomes  from  its  own 
customized  student  assessment  instruments. 

3.1.1  Common  Student  Assessments 

The  research  team  will  administer  project-wide  assessments  to  all  students,  using  the  following  instruments: 

Pre/post  survey:  To  be  administered  at  the  beginning  and  at  the  end  of  capstone  coursework.  This  survey  will  be  used  to  assess 
students’  baseline  understanding  and  changes  as  a  result  of  participating  in  a  capstone  course.  It  will  include  questions  that  allow  the 
students  to  demonstrate  that  they: 

•  Understand  what  systems  engineering  is 

•  Understand  what  systems  engineers  do 

•  Understand  the  qualities  and  skills  that  systems  engineers  bring  to  projects 

•  Are  considering  a  career  in  systems  engineering 

The  specific  questions  pertaining  to  this  survey  are  still  under  review.  The  most  up-to-  date  version  can  be  found  at  the  following  link: 
http://www.survevmonkev.eom/s/serc01 
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Students  Blogs  (Weblogs):  Each  student  in  a  capstone  course  will  be  required  to  post  weekly  to  a  blog,  addressing  the  following 
questions: 

•  What  did  you  and  your  group  accomplish  this  week? 

•  Which  systems  engineering  competencies  best  align  with  what  you  did  this  week?  [See  Appendix  B,  pp.  26  ] 

•  What  specifically  did  you  do  in  terms  of  each  of  the  competencies  you  checked? 

Use  of  weekly  student  blogs  is  to  gauge  the  extent  to  which  students  have: 

•  Developed  and  practiced  systems  engineering  competencies 

At  the  end  of  the  capstone  course  students  will  be  required  to  post  to  a  final  blog,  addressing  the  following  questions: 

•  What  were  the  most  important  system  level  tradeoffs  you  had  to  consider  during  this  project? 

•  If  you  were  to  start  this  project  over  again  what  would  you  do  differently? 

The  final  blog  post  seeks  to  elucidate  student  mastery  of: 

•  Understanding  what  systems  engineers  do 

•  Understanding  how  systems  engineers  think  (analytical  skills) 

Blog  posts  will  be  accessible  to  DoD  mentors  and  experts,  as  well  as  to  the  research  team.  DoD  mentors  and  experts  may  use  the 
blogs  as  one  method  of  communicating  with  the  Capstone  Teams. 
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SE  Case  Study:  Students  will  read  a  brief  case  study  of  the  Bradley  Fighting  Vehicle,  supplemented  with  15  minutes  of  video  from 
the  feature  film  The  Pentagon  Wars,  available  on  YouTube,  after  which,  they  will  respond  to  the  following  prompt: 

•  How  might  a  systems  engineer  have  approached  the  development  of  the  Bradley  Fighting  Vehicle? 

Students  will  complete  the  SE  Case  Study  as  both  as  a  pre/post  assessment,  before  and  after  the  course.  The  objective  of  the  case  study 
analysis  is  to  understand  students’  baseline  understanding  of  what  systems  engineers  do  and  how  systems  engineers  think  (analytical 
skills)  and  to  see  the  changes  in  understanding  that  have  occurred  as  a  result  of  the  course. 

3.1.2  Faculty  Assessment 

In  addition  to  the  information  faculty  provide  in  their  project  reports,  the  research  team  will  administer  surveys  to  all  faculty  teaching 
Capstone  Courses.  These  surveys  will  ask  the  faculty  about  the  effectiveness  of  the  course  sequence  and  of  the  capstone  project 
experience  and  will  be  used  for  a  comparative  analysis.  These  specific  evaluation  instruments  are  being  developed. 

3.1.3  Department  of  Defense  Advisor  Surveys 

DoD  advisors  will  be  asked  to  keep  a  monthly  blog  in  which  they  respond  to  the  following  prompt: 

•  In  your  view,  how  are  the  students  that  you  are  advising  progressing?  What  difficulties  are  they  facing? 

They  will  also  be  asked  to  respond  to  a  survey  at  the  end  of  the  project  that  will  ask  them  to  evaluate  the  success  of  their  role  and  to 
offer  suggestions  for  improving  it.  This  survey  is  also  being  developed. 
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Online  collaboration  and  document  sharing  among  the  research  team,  the  sponsor  contacts,  the  service  academy  coordinators,  and, 
ultimately,  among  PI  and  Capstone  Teams  is  being  implemented  using  Sakai,  a  free,  open  source,  educational  software  platform. 

The  Sakai  software  includes  features  such  as  document  distribution,  discussion  forums, 
assignment  uploads,  live  chat,  wikis,  blogs,  etc.  An  assessment  of  Pis’  desired  functionality  for 
this  online  learning  tool  is  currently  in  progress  in  order  to  build  out  these  resources. 

4.2  RT-19  Public  Website 

A  public  website  containing  general  information  about  the  project  for  other  stakeholder  groups,  including  the  SE  education 
community,  other  federal  agencies,  and  other  organizations,  is  being  developed.  Login  access  into  Sakai  from  this  public  website  will 
be  enabled  specifically  for  RT-19  participants.  The  current  draft  web  site  template  can  be  found  at: 

http://www.stevens.edu/provost/test/home.html.  This  site  is  pending  feedback  from  the  sponsor  regarding  graphical  appearance  and 
online  content. 
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Figure  2.  Website  Home  Page-partial  view 
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Appendices 


Contract  Number:  H98230-08-D-01 71 ,  D02,  TT02,  RT#01 9 

Report  No.  SERC-TR-008 
August  2,  2010 


UNCLASSIFIED 

20 


UNCLASSIFIED 


Appendix  A:  Review  Panei  Rubric 

Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering 

REVIEWER  GUIDELINES 

Use  a  1-5  rating  for  each  question,  with  5  being  the  highest  rating  and  1  the 
lowest.  Please  include  written  comments,  as  warranted,  to  highlight 
aspects  of  the  proposal  pertaining  to  each  question. 

Systems  engineering 

1.  How  well  does  this  proposal  address  and  plan  for  student  learning  of  critical  SE  competencies? 
Consider  each  competency  in  terms  of  its  importance  to  SE  education  and  practice  and  the 
depth  to  which  it  has  been  addressed. 

12345 

Comments; 


2.  Is  the  project  well-planned  and  executable  (e.g.,  planning,  implementation,  coordination, 
recruitment,  development  of  course  materials,  reasonable  timeline)? 

12345 

Comments; 


3.  Overall,  how  would  you  rate  this  proposal  in  terms  of  innovating  over  prior  programs  or 

practices  at  this  institution? 

12345 

Comments; 


4.  To  what  extent  does  the  proposal  ensure  that  students  will  develop  real  products  that 
demonstrate  SE  learning? 


Comments; 


1  2  3  4  5 
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Student  learning 

5.  To  what  extent  are  specific  teaching  and  learning  goals/objectives  listed  and  clearly  articulated? 

12345 

Comments; 


6.  To  what  extent  does  the  proposal  identify  specific  methods  of  assessing  student  learningl  For 
example,  are  specific  instruments  listed?  Are  plans  for  developing  assessments  discussed?  Is 
there  sufficient  evidence  that  student  learning  can  be  effectively  measured? 

1  2  3  4  5 

Comments; 


7.  To  what  extent  does  the  proposal  describe  the  Pl/team's  plans  to  evaluate  the  course  and  its 
components,  including  course  materials,  assessments,  delivery  of  course,  and  other  aspects  of 
the  course,  such  as  recruitment,  coordination  among  departments,  etc.? 

1  2  3  4  5 

Comments; 


8.  How  likely  is  this  project  to  increase  student  awareness  of  SE  career  opportunities,  and/or  of  the 
practice  and  role  of  SE  in  the  workplace? 

12345 

Comments; 

9.  What  impact,  if  any,  will  this  project  have  on  advancing  understanding  of  SE  education,  including 
approaches,  models,  etc.?  (Please  provide  a  brief  statement,  if  applicable;  otherwise,  write  n/a 
or  unknown). 
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Other 

10.  How  well-aligned  is  the  proposal  with  one  of  the  required  DOD  problem  areas  (see  Appendix  B 
for  list)? 

12345 

Comments; 


11.  To  what  extent  does  the  team  possess  the  necessary  internal  expertise  and  resources  to 
effectively  implement  the  project? 

12345 

Comments; 

12.  To  what  extent  does  the  team  make  use  of  external  expertise,  such  as  DoD  or  industry  mentors 
or  clients? 


Comments; 


12345 


13.  To  what  extent  does  the  proposal  present  a  strong  plan  for  continuing,  sustaining,  and 
institutionalizing  the  course  after  this  funding  cycle? 

12345 

Comments; 


14.  Does  the  proposal  include  any  plan  for  dissemination? 

YES  NO 


15.  Is  the  budget  reasonable,  well-justified,  and  clearly  aligned  with  project  activities? 


Comments; 


12345 
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Recommendation  for  funding 

High  Medium 

Areas  of  strength: 


Suggestions  for  improvement: 


Other  comments,  if  applicable: 


Initials: 

Date: 
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Appendix  B:  Invitation  to  SERC  Collaborators 
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Appendix  C:  Review  Panei  Summaries 

University  of  Maryland 

Overall  rating:  208  out  of  260,  with  three  High  recommendations  and  one  Medium 

recommendation  for  funding. 

The  panelists  considered  this  to  be  a  good  proposal,  with  clear  goals,  a  strong  team,  and  good  ties 
with  outside  experts.  There  were  concerns  that  there  were  too  many  competencies  addressed  for 
one  course.  Therefore,  please  identify  the  competencies  that  are  the  highest  priority  that  you  will 
measure,  and  a  secondary  set  of  priorities.  Please  provide  additional  details  explaining  how  the 
project  will  impact  graduate  level  education. 

Missouri  University  of  Science  and  Technology 

Overall  rating:  208  out  of  260,  with  three  High  recommendations  and  one  Medium 

recommendation  for  funding. 

The  panelists  felt  that  this  was  a  strong  proposal,  well  thought-out,  with  clearly  articulated  goals 
and  a  detailed  evaluation  plan.  The  panelists  noted  several  concerns  that  should  be  addressed  in 
the  updated  work-plan  and  timeline.  Provide  assurances  that  the  budget  includes  the  funds 
needed  to  purchase  the  necessary  materials.  Also,  please  provide  a  description  of  how  the  course 
will  be  institutionalized,  given  the  very  specific  nature  of  the  deliverables 

Stevens  Institute  of  Technology 

Overall  rating:  204  out  of  260,  with  two  High  recommendations  and  two  Medium 

recommendations  for  funding. 

The  panelists  felt  that  this  proposal’s  strengths  were  its  focus  on  a  small  number  of  competencies 
and  its  evaluation  plan.  Some  parts  of  the  proposal  lacked  specifics,  and  that  therefore  it  was 
difficult  to  be  sure  that  this  would  be  a  replicable  model. 
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There  were  concerns  that  no  DoD  subject  matter  experts  were  named,  that  the  role  of  the 
workshops  was  unclear.  Please  provide  additional  details  about  these  concerns  as  well  as  the 
plans  for  the  monthly,  “all  hands”  workshops. 

University  of  Virginia 

Overall  rating:  204  out  of  260,  with  two  High  recommendations  and  two  Medium 
recommendations  for  funding. 

The  panelists  felt  that  this  proposal  was  strong  in  that  it  focused  on  expanding  SE  to  non-systems 
engineering  students.  There  were  concerns  that  too  many  competencies  were  addressed.  The 
panelists  were  also  unclear  whether  a  course,  a  project  within  a  course  or  seminar  was  being 
offered.  Please  provide  a  more  detailed  explanation  of  the  plan  for  the  course  or  seminar  that  will 
be  conducted  and  of  how  the  course  will  be  institutionalized. 

Pennsylvania  State  University 

Overall  rating:  202  out  of  260,  with  three  High  recommendations  and  one  Medium 
recommendation  for  funding. 

The  panelists  felt  this  was  a  strong  proposal  because  of  its  interdisciplinary  aspects.  The  panelists 
were  concerned  that  too  many  competencies  were  addressed  for  a  one  semester  course.  Some 
reviewers  were  concerned  that  one  course  would  not  be  enough  to  introduce  students  to  SE  and 
fabricate  a  prototype  as  well.  Please  provide  a  more  detailed  description  for  how  SE  will  be 
addressed  and  the  level  to  which  it  will  be  addressed  within  the  one-semester  course,  and/or  a 
justification  for  conducting  this  effort  as  a  one-semester  course.  Address  reviewers’  concerns 
that  the  SE  competencies  along  with  a  prototype  can  be  developed  in  the  context  of  a  one- 
semester  course. 

Southern  Methodist  University 

Overall  rating:  201  out  of  260,  with  two  High  recommendations  and  two  Medium 
recommendations  for  funding. 
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The  panelists  considered  this  proposal  strong  because  of  the  client  and  the  access  to  the  Skunk 
Works  IDE  that  requires  students  to  participate  in  client  and  risk  driven,  real  world  challenges. 
Reviewers  expressed  concerns  that  it  may  not  be  possible  to  address  in  depth  all  the  SE 
competencies  listed.  Therefore,  please  identify  the  competencies  that  are  the  highest  priority  that 
you  will  measure,  and  a  secondary  set  of  priorities.  Please  provide  a  more  detailed  explanation 
about  the  need  for  capital  equipment,  non-capital  equipment  and  foreign  travel. 

Auburn  University 

Overall  rating:  200  out  of  260,  with  two  High  recommendations  and  two  Medium 

recommendations  for  funding. 

The  panelists  felt  this  was  a  good  proposal.  Areas  of  strength  noted  by  the  panelists  included  the 
bridging  of  two  departments,  the  strong  team  and  the  involvement  of  the  Biggio  Center. 

The  panelists  noted  several  concerns  that  should  be  addressed  in  the  updated  workplan  and 
timeline.  The  proposed  course  addresses  many  competencies.  PI  is  requested  to  specify  the  most 
critical  competencies  and  the  degree  to  which  they  will  be  addressed.  Plans  for  student 
recruitment  and  institutionalization  need  to  be  described  in  greater  detail. 

Wayne  State  University 

Overall  rating:  198  out  of  260,  with  two  High  recommendations  and  two  Medium 

recommendations  for  funding. 

The  panelists  liked  the  idea  of  introducing  modules  into  various  courses,  which  they  felt  was  a 
different  and  feasible  approach  to  expanding  SE  concepts,  and  the  focus  on  a  limited  number  of 
SE  competencies.  There  were  suggestions  that  the  dissemination  plan  needs  strengthening  and 
that  it  was  not  clear  how  the  external  experts  would  contribute  to  the  course  modules.  Please 
address  these  concerns  in  your  project  plans  and  provide  a  response. 

Purdue  University  (not  recommended  for  funding) 

Overall  rating:  174  out  of  260,  with  no  High  recommendations  and  4  Medium  recommendations 
for  funding. 
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The  panelists  felt  that  this  proposal  did  not  clearly  articulate  the  teaching  and  learning  objectives 
for  the  course  and  that  too  many  competencies  were  addressed.  The  proposal  is  a  moderate 
improvement  of  a  previous  existing  course  but  it  doesn’t  constitute  an  important  innovative 
approach  to  teach  Systems  Engineering.  It  includes  a  strong  research  team  and  shows  potential 
in  the  initial  phase  of  definition  of  requirements;  however,  it  doesn’t  clearly  address  the 
development  of  real  products  by  students. 

University  of  Alabama  (not  recommended  for  funding) 

Overall  rating:  162  out  of  260,  with  no  High  recommendations  and  2  Medium  recommendations 
for  funding. 

From  the  review  panel  perspective  Systems  Engineering  learning  objectives  in  this  proposal  are 
not  clearly  established.  In  other  words,  although  a  detailed  plan  is  shown  and  a  strong 
multidisciplinary  team  is  involved  the  SE  educational  content  remains  vague.  The  topic  is 
aligned  with  DOD  focus  areas;  however,  traceability  to  competencies  of  interest  (Appendix  A)  is 
not  evident  at  all. 

University  of  Southern  California  (not  recommended  for  funding) 

Overall  rating:  162  out  of  260,  with  no  High  recommendations  and  2  Medium  recommendations 
for  funding. 

The  panelists  gave  this  proposal  the  lowest  score.  There’s  no  significant  innovation  over  prior 
programs  as  already  existing  courses  are  used.  Although  an  experienced  PI  is  involved,  the 
proposal  itself  is  weak.  There’s  no  evidence  of  competencies  addressed,  not  even  a  detailed  plan 
was  provided. 
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Appendix  D:  Summary  of  Course 
Implementation/Revisions 


Institution 

# 

students 

Involved 

DoD  problem 

area(s) 

targeted 

(numerical  code) 

*Competencies 

Addressed 

(numerical  code) 

Term  of 
Course 

Auburn 

University 

40 

1 

Primary:  12 

Secondary:  1,  2,  4,  5,  6,  7,  8,  9, 10, 11, 

13, 14, 15, 16, 17, 18, 19,  20,  21,  22,  23, 
24,  25,  26,  27,  28,  29 

Two 

sequential 

courses: 

Fall  2010 
Spring  2011 

Missouri 
Science  & 
Technology 

30 

5 

1,  2,  3, 4,  5,  6,  8,  9, 10, 12, 13, 14, 15, 

16, 17, 18, 19,  20,  21,  22,  23,  24,  26,  27, 
28,  29 

Two 

sequential 

courses: 

Fall  2010 
Spring  2011 

Penn  State 
University 

25-35 

2,  3,  4,  5 

Primary:  4,  5,  6,  8,  9, 10, 14, 15, 19,  26, 
27 

Secondary:  2, 16, 18,  22,  24,  25,  29 

One  Course: 
Fall  2010 

Southern 

Methodist 

University 

73 

5 

Primary:  4,  5,  6,  8, 18,  21,  22,  26,  27 

Secondary:  2,  7,  9, 10, 13, 14, 17, 19, 

25,  29 

Fall  2010 

Spring  2011 

Stevens 
Institute  of 
Technology 

17  -  22 

4 

Primary:  2,  4, 5,  6,  8,  9, 10, 19,  21,  26 
Secondary:  1,  7, 18,  24 

Fall  2010 
Spring  2011 

University 
of  Virginia 

18  -  20 

3,5 

1,  2,  3, 4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 
15, 16, 18, 19,  21,  22,  26,  27,  28,  29 

Fall  2010 
Spring  2011 

Institution 

# 

students 

involved 

DoD  problem 

area(s) 

targeted 

(numerical  code) 

*Competencies 

Addressed 

(numerical  code) 

Term  of 
Course 

Wayne  State 
University 

80  - 100 

3 

2, 4,  6,  8, 14, 17, 18,  20 

Fall  2010 
Spring  2011 

University 
of  Maryland 

46 

1,2 

Primary:  4,5,6,24 

Secondary:  8,  9  ,10, 14 

Fall  2010 
Spring  2011 

Air  Force 
Institute  of 

3-6 

2 

2,  3, 4,  5,  6,  7,  8,  9, 10 

Fall  2010 
Spring  2011 
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Technology 

US  Air 

Eorce 

Academy 

6 

2,3 

1,  3,  5,  6,  7,  8,  9, 10, 11, 15,  i6,  i8, 19, 

21,  22,  23,  26,  27,  29 

Pall  2010 
Spring  2011 

US  Military 
Academy  at 
West  Point 

4 

5 

No  information  available. 

Pall  2010 
Spring  2011 

^Appendix  A,  SPRDE-SE/PSE  Competency  Model 


NOTES: 

1.  All  information  pertaining  to  NPS  is  still  pending 

2.  Over  350  students  impacted  by  this  pilot  program. 

3.  The  most  targeted  DoD  problem  areas  are:  2,  3,  5 

4.  The  addressed  competencies  were  revised  by  corresponding  Pis  of  each  partner  institution 
after  receiving  comments  from  the  panel  review. 

5.  Interim  reports  of  each  partner  institution  contain  detailed  information  of  their  own  pilot 
programs.  These  reports  are  available  for  review  in  Sakai  at  the  following  link: 
https://gatewav.stevens.edu/portal. 
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Appendix  E:  Course  Implementation  Program 

Timeline 
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Appendix  F;  Evaluation  Framework,  Assessments 

and  Data  Sources* 

*As  of  July  20,  2010 


STUDENTS 

Objectives: 

At  the  end  of  the  capstone 
course,  students  should: 

Possible  questions: 

Instrument 

Understand  what  systems 
engineering  is 

What  is  systems  engineering?  Define  it  as  best  you  can. 

Pre/post  survey 

In  the  context  of  systems  engineering,  give  an  exampie  of  a 
system  that  wouid  be  addressed  by  a  systems  engineer. 

Pre/post  survey 

Understand  what  systems 
engineers  do 

How  might  systems  engineers  differ  from  discipiinary  engineers 
(mechanicai,  structurai,  etc.)  working  on  a  muitidiscipiinary  team? 
Give  an  exampie. 

Pre/post  survey 

Understand  the  qualities  and 
skilis  that  systems  engineers 
bring  to  projects 

List  five  words  that  describe  the  skiiis  and  quaiities  needed  by  a 
systems  engineer. 

Pre/post  survey 

Consider  a  career  in  systems 
engineering 

On  a  scaie  of  1  to  5,  with  5  being  the  highest  (Not  at  aii  to  Very 
much),  how  interested  are  you  in  becoming  a  systems  engineer? 
[Inciude  Don't  know/Not  sure] 

Pre/post  survey 

Post  oniy: 

On  a  scaie  of  1  to  5,  with  5  being  the  highest  (Not  at  aii  to  Very 
Much),  how  much  did  this  project  make  you  want  to  expiore 
systems  engineering  further? 

Deveiop  and  practice  the  skiiis 
of  systems  engineers 

What  did  you  and  your  group  accompiish  this  week? 

Weekiy  blogs 

Which  systems  engineering  competencies  best  aiign  with  what  you 
did  this  week?  [Choose  from  iist] 

Weekly  blogs 

What  specificaliy  did  you  do  in  terms  of  each  of  the  competencies 
you  checked? 

Weekly  blogs 

Understand  what  systems 
engineers  do/how  systems 
engineers  think  (anaiytic  skiiis) 

What  were  the  most  important  system-ievei  trade-offs  you  had  to 
consider  during  this  project? 

Final  blog  post 

If  you  were  to  start  this  project  over  again,  what  wouid  you  do 
differentiy? 

Final  blog  post 

Understand  what  systems 
engineers  do/how  systems 
engineers  think  (anaiytic  skiiis) 

How  wouid  a  systems  engineer  have  approached  the  probiem 
described  in  this  scenario? 

[Pre  version  has  individuai  response,  foiiowed  by  group  discussion, 
foiiowed  by  revisit  of  original  post;  post-version  has  one  individuai 
response  oniy] 

Pre/post 

scenario 
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FACULTY 

Create  an  effective  course 
sequence 

To  what  extent  did  the  courses  you  created  give  students  an 
understanding  of  what  systems  engineering  is  and  what  systems 
engineers  do? 

Survey/interview 

How  was  SE  iearning  integrated  into  the  capstone  sequence?  What 
methods  did  you  use  (iecture,  etc.)  and  how  did  you  time  when 
the  competencies  wouid  be  introduced? 

Survey/interview 

How  do  you  know?  What  are/were  your  criteria  forjudging  this? 

Survey/interview 

If  this  was  a  two-course  sequence,  did  the  first  course  sufficientiy 
prepare  the  students  for  the  capstone  project? 

Survey/interview 

Create  an  effective  capstone 
project  experience 

On  a  scaie  of  1  to  5,  with  5  being  the  highest,  how  effective  was 
the  reiationship  with  the  ciient?  Piease  describe  the  benefits  and 
the  issues  (if  any). 

Survey/interview 

Overaii,  how  weii  did  the  students'  finai  projects  meet  your 
iearning  objectives? 

Survey/interview 

How  did  you  assess  them?  What  were  your  criteria? 

Survey/interview 

What  wouid  you  do  differentiy  the  next  time  around? 

Survey/interview 

How  do  you  pian  to  institutionaiize  this  sequence? 

Survey/interview 

DOD  ADVISORS 

Effectiveiy  consuit  with  one  or 
more  groups  of  students 

In  your  view,  how  are  the  students  you  are  advising  progressing? 
What  difficuities  are  they  having? 

Monthiy  iog 

Overaii,  did  you  feei  that  your  advice  was  usefui  to  the  group  and 
that  the  students  took  advantage  of  it? 

Survey/interview 

Did  you  feei  that  the  projects  were  successfui?  If  so,  what  was 
successfui  about  them?  If  not,  why  not? 

Survey/interview 

How  couid  the  roie  of  the  DoD  Advisor  have  been  better  defined  or 
impiemented  to  improve  the  process  in  the  future? 

Survey/interview 
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